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£j i Abstract 

Efficient online learning with pairwise loss functions is a crucial component in building large- 
scale learning system that maximizes the area under the Receiver Operator Characteristic 
(ROC) curve. In this paper we investigate the generalization performance of online learning 
algorithms with pairwise loss functions. We show that the existing proof techniques for 
generalization bounds of online algorithms with a univariate loss can not be directly applied 
to pairwise losses. In this paper, we derive the first result providing data-dependent bounds 
c/2 | for the average risk of the sequence of hypotheses generated by an arbitrary online learner in 

terms of an easily computable statistic, and show how to extract a low risk hypothesis from 
the sequence. We demonstrate the generality of our results by applying it to two important 
problems in machine learning. First, we analyze two online algorithms for bipartite ranking; 
one being a natural extension of the perceptron algorithm and the other using online convex 
optimization. Secondly, we provide an analysis for the risk bound for an online algorithm 
iy ^ | for supervised metric learning. 

Keywords: Generalization bounds, Pairwise loss functions, Online learning, Loss bounds, 
, Bipartite Ranking, Metric Learning 
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1. Introduction 

The standard framework in learning theory considers learning from examples Z n = {(xt,yt) 
£ X x y} 7 t = 1, 2, • • • , n, (independently) drawn at random from an unknown probability 
distribution T> on Z := X x y (e.g. X = JR^ and y = IR). Typically a univariate loss 
function £(h, (x, y)) is adopted to measure the performance of the hypothesis h : X — > y, for 
example, £(h, (x,y)) = (h(x) — y) 2 for regression or £(h,x,y) = Hr^^Wyi for classification. 
The aim of learning is to find a hypothesis that generalizes well, i.e. has small expected 
risk ~R^(x,y )t(h, (x,y)). 

In this paper we study learning in the context of pairwise loss functions, that depend on 
pairs of examples and can be expressed as £ (h, (x, y), (u, v)) where the hypothesis is applied 
to pairs of examples, i.e. h : X x X — > IR. Pairwise loss functions capture ranking problems 
that are important for a wide range of applications. For example, in the supervised ranking 
problem one wishes to learn a ranking function that predicts the correct ordering of objects. 
The hypothesis h is called a ranking rule such that h(x, u) > if x is ranked higher than u 
and vice versa. The misranking loss (Clemengon et al., 2008; Peel et al., 2010) is a pairwise 
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loss such that 

4ank 0, (X, y), (W, «)) = I[( 2/ -«)(/ l ( a! , u ))<0] , 

where E is the indicator function and the loss is 1 when the examples are ranked in the 
wrong order. The goal of learning is to find a hypothesis h that minimizes the expected 
misranking risk 1Z(h), 

K(h) := E^E^ [£ (h, (x, y), (u, v))] . (1) 

In many interesting cases, finding a ranking rule amounts to learning a good scoring function 
s : X — > IR such that h{x,u) = s(x) — s(u). Therefore, higher ranked examples will have 
higher scores. Another application comes from distance metric learning, where the learner 
wishes to learn a distance metric such that examples that share the same label should be 
close while ones from different labels are far away from each others. 

This problem, especially the bipartite ranking problem where y = {+1, — 1}> has been 
extensively studied over the past decade in the batch setting, i.e., where the entire sequence 
Z n is presented to the learner in advance of learning. Freund et al. (2003) gave general- 
ization bounds for the RankBoost algorithm, based on the uniform convergence results for 
classification. Agarwal et al. (2005) derived uniform convergence bounds for the bipartite 
ranking loss, using a quantity called rank-shatter coefficient, which generalizes ideas from 
the classification setting. Agarwal et. al. provided bounds for the bipartite ranking problem 
(Agarwal and Niyogi, 2005) and the general ranking problem (Agarwal and Niyogi, 2009) 
using ideas from algorithmic stability. Rudin et al. (2005) approached a closely related 
problem where the goal is to correctly rank only the top of the ranked list, and derived 
generalization bounds based on covering number. Recently, several authors investi- 
gated oracle inequalities for pairwise-based quantities via the formalization of [/-statistics 
(Clemengon et al., 2008; Rejchel, 2012) using empirical processes. Peel et al. (2010) gave 
an empirical Bernstein inequality for higher order [/-statistics. Another thread comes 
from the perspective of reducing ranking problems to the more familiar classification prob- 
lems (Kotllowski et al., 2011; Ertekin and Rudin, 2011; Agarwal, 2012). 

In this paper we investigate the generalization performance of online learning algo- 
rithms, where examples are presented in sequence, in the context of pairwise loss functions. 
Specifically, on each round t, an online learner receives an instance Xt and predicts a label yt 
according to the current hypothesis h t -\. The true label yt is revealed and ht-i is updated. 
The goal of the online learner is to minimize the expected risk w.r.t. a pairwise loss function 
£. 

Over the past two decades, online learning algorithms have been studied extensively, 
and theoretical results provide relative loss bounds, where the online learner competes 
against the best hypothesis (with hindsight) on the same sequence. Conversions of online 
learning algorithms and their performance guarantees to provide generalization performance 
in the batch setting have also been investigated (e.g., (Kearns et al., 1987; Littlestone, 
1990; Freund and Schapire, 1999; Zhang, 2005)). Cesa-Bianchi et al. (2004) provided a 
general online-to-batch conversion result that holds under some mild assumptions on the 
loss function. Given a univariate loss function I, a sample Z n and an ensemble of hypotheses 
{hi,h2,"' ,h n } generated by an online learner A, the following cumulative loss of A is 
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defined as 



1 

M n = M n {Z n ) = -J2Kht-i,Z t ). 



t=i 



Cesa-Bianchi and Gentile (2008) showed (as a refined version of the bound in (Cesa-Bianchi et al., 
2004)) that one can extract a hypothesis h from the ensemble such that 



Therefore, if one can develop an online learning algorithm with bounded cumulative loss 
for every possible realization of Z n , then its generalization performance is guaranteed. 
A sharper bound exists when the loss function is strongly convex (Kakade and Tewari, 
2009). The key step of these derivations is to realize that Vt-\ = 1Z{h t -\) — £(hf-i, Z t ) is 
a martingale difference sequence. Thus one can use martingale concentration inequalities 
(Azuma's inequality or Friedman Inequality) to bound J^Vt- Unfortunately, this property 
no longer holds for pairwise loss functions. 

Of course, as mentioned for example in the work of Peel et al. (2010, Sec. 4.2), one can 
slightly adapt an existing online learning classification algorithm (e.g., perceptron), feeding 
it with data sequence z t := (z2t-i,Z2t) and modifying the update function accordingly. In 
this case, previous analysis (Cesa-Bianchi and Gentile, 2008) does apply. However, this does 
not make full use of the examples in the training sequence. In addition, empirical results 
show that this naive algorithm, which corresponds to the algorithm for online maximization 
of the area under the ROC curve (AUC) with a buffer size of one in (Zhao et al., 2011), is 
inferior to algorithms that retain some form of the history of the sequence. Alternatively, 
it is tempting to consider feeding the online algorithm with pairs z\ = (zi,Zt),i < t on 
each round. However, in this case, existing results would again fail because z\ are not i.i.d. 
Hence, a natural question is whether we can prove data dependent generalization bounds 
based on the online pairwise loss. 

This paper provides a positive answer to this question for a large family of pairwise 
loss functions. On each round t, we measure Mj, the average loss of ht-x on examples 
(zj, Zt), i < t. Let A4 n denote the average loss, averaging Mt over t ^ (1 — c)n on a training 
sequence of length n where c is a small constant. The main result of this paper, provides 
a model selection mechanism to select one of the hypotheses of an arbitrary online learner, 
and states that the probability that the risk of the chosen hypothesis h satisfies, 



Here J\f(rl,rj) is the covering number for the hypothesis class T~L and Lip((f>) is determined 
by the Lipschitz constant of the loss function (definitions and details are provided in the 
following sections). Thus, our results provide an online-to-batch conversion for pairwise loss 
functions. We demonstrate our results with the following two applications: 




n[h) ^M n + e 



is at most 
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1. We analyze two online learning algorithms for the bipartite ranking problem. We first 
provide an analysis of a natural generalization of the perceptron algorithm to work 
with pairwise loss functions, that provides loss bounds in both the separable case and 
the inseparable case. As a byproduct, we also derive a new simple proof of the best L\ 
based mistake bound for the perceptron algorithm in the inseparable case. Combining 
with our main results we provide the first online algorithm with corresponding risk 
bound for bipartite ranking. Secondly, we analyze another algorithm using the online 
convex optimization techniques, with similar risk bounds. 

2. Several online metric learning algorithms have been proposed with corresponding re- 
gret analyzes, but the generalization performance of these algorithms has been left 
open, possibly because no tools existed to provide online-to-batch conversion with 
pairwise loss functions. We provide risk bounds for an online algorithm for distance 
metric learning combining with the results for online convex optimization with matrix 
argument. 

The rest of this paper is organized as follows. Section 2 defines the problem and states 
our main technical theorem and Section 3 provides a sketch of the proof. We provide 
model selection results and risk analysis for convex and general loss functions in Section 4. 
In Section 5, we describe our online algorithm for bipartite ranking and analyze it. The 
results in sections 2-5 are given for a model and algorithms with an "infinite buffer", that 
is, where the update of the online learner at step t depends on the entire history of the 
sequence, z\,--- ,z%-\. Section 6 shows that the results and algorithms can be adopted 
to a buffer of limited size. Interestingly, to guarantee convergence our results require that 
the buffer size grows logarithmically with the sequence size. Section 7 is devoted to the 
analysis of online metric learning. Finally, we conclude the paper and discuss possible 
future directions in Section 8. 

2. Main Technical Result 

Given a sample Z n = {z\, • • • , z n } where Zi=(xi,yi) and a sequence of hypotheses ho, hi, ■ ■ ■ , 
h n generated by an online learning algorithm, we define the sample statistic M n as 



where c n = \c ■ n\ and c 6 (0, 1) is a small positive constant. Mt(Z t ) measures the perfor- 
mance of the hypothesis on the next example Zt when paired with all previous examples. 
Note that instead of considering all the n generated hypotheses, we only consider the av- 
erage of the hypotheses h Cn -\, • • • , h n -2 where the statistic Mt is reliable and the last two 
hypotheses h n _\,h n are discarded for technical reasons. In the following, to simplify the 
notation, M n denotes M n (Z n ) and M t denotes M t (Z l ). We define / A g = min(/, g) and 
/ V g = max(/,#). 

As in (Cesa-Bianchi et al., 2004), our goal is to bound the average risk of the sequence 
of hypotheses in terms of Ai n , which can be obtained using the following theorem. 
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Theorem 1 Assume the hypothesis space (H, \\ ■ ||oo) is compact. Let ho,hi,--- ,h n G H 
be the ensemble of hypotheses generated by an arbitrary online algorithm working with a 
pairwise loss function £ such that, 

£(h,zi,z 2 ) = 4>{yi - y 2 ,h(x 1 ,x 2 )), 

where <f> : 3^ x y —> [0,1] is a Lipschitz function w.r.t. the second variable with a finite 
Lipschitz constant Lip(<p). Then, Vc > 0, Ve > 0, we have for sufficiently large n 



n ^ £ *(*_,) > M- + e I < [W («, jj^) + l] exp {-^^ + Inn} 



(3) 



Here the covering number J\f(H, rj) is denned to be the minimal I in IN such that there 
exist I disks in % with radius r/ that cover T~L. We make the following remarks. 

Remark 2 Let JEt denote JE Zt [-\zi, ■ ■ ■ ,Zt-\\. It can be seen that IE^M^] — lZ(ht—\) is no 
longer a martingale difference sequence. Therefore, martingale concentration inequalities 
that are usually used in online-to -batch conversion do not directly yield the desired bound. 

Remark 3 We need the assumption that the hypothesis space T~L is compact so that its 
covering number J\f(H, rj) is finite. As an example, suppose X C IR d and the hypothesis space 
is the class of linear functions that lie within a ball B^(lR d ) = {w 6 lR d : sup (to, a;) ^ R}. 

It can be shown (see Cucker and Zhou, 2007, chap. 5) that the covering number is one if 
rj > R and otherwise 

M(B R ,r,)^(^ + l\ . (4) 

Remark 4 We say that f(s,t) is Lipschitz w.r.t the second argument if Vs,\f(s,ti) — 
f(s,t 2 )\ ^ Lip(f)\\t\ — t 2 \\. This form of the pairwise loss function is not restrictive and is 
widely used. For example, in the supervised ranking problem, we can take the hinge loss as 

hinge(h, z x ,z 2 ) = <j>{yi - y 2 , h(xi) - h(x 2 )) = [1 - {h{xij - h(x 2 ))(yi - y 2 )] + , 

which can be thought as a surrogate function for £ ran k- Since 4> is not bounded, we define 
the bounded hinge loss using (p(s, t) = min([l — st] + , 1) 6 [0, 1] i/s / and otherwise. We 
next show that (f> is Lipschitz. This is trivial for y = 0. For y ^ 0, when the first argument 
is bounded by a constant C, (p(y, ■) satisfies 

\<f>(y,xi) - 4>{y,x 2 )\ ^ | [1 - yxi] + - [1 - yx 2 ] + | < \\yx x - yx 2 \\ C\\xi - x 2 \\. 

Alternatively, one can take the square loss, i.e. £(h, z\, z 2 ) = [1 — (h(xi) — h(x 2 ))(yi — y 2 )] 2 ■ 
If its support is bounded then I is Lipschitz. 
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3. Proof of the Main Technical Result 

The proof is inspired by the work of (Cucker and Smale, 2002; Agarwal et al., 2005; Rudin, 
2009). The proof makes use of the Hoeffding-Azuma inequality, McDiarmid's inequality, 
symmetrization techniques and covering numbers of compact spaces. 
Proof [Proof of Theorem 1] By the definition of Ai n (see (2)), we wish to bound 



which can be rewritten as 

n-l 



n — c 



1 n— 1 1 n— 1 

— y, nht-i) Y, Mt > 



t=Cn 



(5) 



t=C n 



n — o 



i " x 



t=c„ 



K{h 



t-ij 



E t [M t ; 



+ 



n — o 



1 



t=c„ 



n — o 



r n— 1 



t=Cn 



TZ(h 



t-i. 



>2 + 



m t [M t ]-M t 



> e 



Tl — On 



t=c„ 



m t [M t ]-M t 



(6) 



Thus, we can bound the two terms separately. The proof consists of four parts, as follows. 



Step 1: Bounding the Martingale difference 

First consider the second term in (6). We have that Vj = (Et[Aft] — Mt)/{n — c n ) is a 
martingale difference sequence, i.e. EtfVf] = 0. Since the loss function is bounded in [0, 1], 
we have \Vt\ ^ l/(n — c n ), t = 1, • • • , n. Therefore by the Hoeffding-Azuma inequality, 
J2t Vt can be bounded such that 



n — o 



-. n— 1 

-Y. 



t=c n L 



IB t [M t ] - Af t 



> 2 < exp 



1 - c)ne 2 



(7) 



Step 2: Symmetrization by a ghost sample E n 

In this step we bound the first term in (6). Let us start by introducing a ghost sample 
H n = = {(xj,yj)},j = 1, • • • , n where each £j follows the same distribution as Zj. 
Recall the definition of Aff and define Aff as 



t 



. t-i j t-i 

— Y^t-i^t^j), M t = — y^^(/tt_a,zt,^). 



(8) 



The difference between Aft and Aft is that Aft is the sum of the loss incurred by /it— l on 
the current instance Zt and all the previous examples Zj, j = 1, • • • ,t — 1 on which ht~\ is 
trained, while Aff is the loss incurred by the same hypothesis ht—i on the current instance 
Zt and an independent set of examples £j, j = 1, • • • , t — 1. 
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Claim 1 The following equation holds 

(1 i—l \ / i n— 1 
V [ft(/i t _i) - Et[M t ]] ^ e < 2P Z ^ P ™ V \jE t [M t ] - E t [M t ] 



t=c„ 



(9) 



whenever n > 2/(e c ). 



Notice that the probability measure on the right hand side of (9) is on Z n x H n . 

Proof [Sketch of the proof of Claim 1] It can be seen that the RHS (without the factor of 

2) of (9) is at least 



■ z n ~T>" 
H"~D" \ n — Cr, 



— V [n{h t - 1 ) - w, t \M t ]} ^ fl V [Et[Aft] - n(h t -i 



ftt-i) 



Since lE-^xm IEtfMj = 7Z(h t -i), by Chebyshev's inequality 



n - c. 



n-l 

£ 

t=Cn 



1E t [M t )-K(h t -i) 



Z n > 



1 e 2 /4 

(10) 



To bound the variance, we first investigate the largest variation when changing one random 
variable £j with others fixed. From (8), it can be easily seen that changing any of the £j 
varies each IEi[Mi], where t > j by at most by l/(t — 1). Recall that we are only concerned 



with IEj [Mj] when t ^ c n . Therefore, we can see that the variation of — — Ylt=c ^t[Mi 
regarding the jth example £» is bounded by 



1 



n — o„ 



n—1 -. 

y — 

^ t-i 



i 



n — On 



_t=(jVc n +l) 

Thus, by Theorem 9.3 in (Devroye et al., 1996), we have 

n— 1 



n—1 -, 



.t=c„+l 



cn 



(11) 



(12) 



t=Cn 



1=1 



Thus, whenever e 2 c 2 n > 2, the LHS of (10) is greater or equal than 1/2. This completes 
the proof of Claim 1. ■ 



Step 3: Uniform Convergence 

In this step, we show how one can bound the RHS of (9) using uniform convergence tech- 
niques, McDiarmid's inequality and covering number. Our task reduces to bound the 
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following quantity 



n 



1 n— 1 

[lE t [M t ] - Et[M t ] 

" ^ t=Cn 



(13) 



Here we want to bound the probability of the large deviation between the empirical per- 
formance of the ensemble of hypotheses on the sequence Z n on which they are learned 
and on an independent sequence E n . Since h t relies on zi,--- , z t and is independent 
°f we resort to uniform convergence techniques to bound this probability. Define 

L t (h t -i) = E t [M t ] - TE t [M t ]. Thus we have 



1 



n—l 



n 



t — Cn 



j s$p 


sup 




\h Cn ,--- ,h n -i 



1 



n-l 



n 



H-lj 



t=c 



> e 



n-l 



^2 P£*~£>*,S*~X>* SU P 



Lt(fc)l > e I . (14) 



t=c 



To bound the RHS of (14), we start with the following lemma. 
Lemma 5 Given any function f G % and any t ^ 2 

Pz*~2>t lS *~2>* OW) > e) < exp {-(t - l)e 2 } . 



(15) 



The proof which is given in the appendix shows that Lt(f) has a bounded variation of l/(t— 

1) when changing each of its 2{t — 1) variables and applies McDiarmid's inequality. Finally, 

our task is to bound P(sup [L t (f)] ^ e). Consider the simple case where the hypothesis space 
f&H 

% is finite, then using the union bound, we immediately get the desired bound. Although 
% is not finite, a similar analysis goes through based on the assumption that T~L is compact. 
We will follow Cucker and Smale (2002) and show how this can be bounded. The next two 
lemmas (see proof of Lemma 6 in the appendix) are used to derive Lemma 8. 

Lemma 6 For any two functions hi,h,2 £ T~L, the following equation holds 

L t {h x ) - L t (h 2 ) ^ 2Lip{<f,)\\hi - Moo- 
Lemma 7 Let T-i = Si U • • • U Si and e > 0. Then 



P (supL t (h) > < VF ( sup L t (h) ^ e] 
\hen J fr[ \heSj J 



Lemma 8 For every 2 ^ t ^ n, we have 



(jup[L t (h)} 2e\<M\n, 



ALip((f>) 



exp 



(t - 1)6 2 



(16) 
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Proof [Proof of Lemma 8] Let I = N [%, 2 Lip(0) j an< ^ consider hi, ■ ■ ■ ,hi such that the 
disks Dj centered at hj and with radius 2 Lip(» cover By Lemma 6, we have 

\Lt(h) - L t (hj)[ < 2Lip(0)||/t - hjWoc < e. 

Thus, we get 

P ( sup L t (/i) ^ 2e ) < P (Z^fy) ^ e) 

\hEDj J 

Combining this with (15), and Lemma 7 and replacing e by e/2, we have (16). ■ 
Combining (16) and (14), we have 



n L_g it(At _ l)>e j <v („,_£_)„^|_(s^d}. (17) 



This shows why we need to discard the first c n hypotheses in the ensemble. If we include 
/i2 for example, according to (16), we have P(Z/2(/) ^ e) ^ . As n grows, this heavy 
term remains in the sum, and the desired bound cannot be obtained. 

Step 4: Putting it all together 

From (9) and (14) and substituting e with e/4 in (17), we have 

(c„ - l)e 2 



G4 



(18) 

From (18) and (7), accompanied with the fact that (18) decays faster than (7), we complete 
the proof for Theorem 1. ■ 



4. Model Selection 

Following Cesa-Bianchi et al. (2004) our main tool for finding a good hypothesis from the 
ensemble of hypotheses generated by the online learner is to choose the one that has a small 
empirical risk. We measure the risk for h% on the remaining n—t examples, and penalize each 
ht based on the number of examples on which it is evaluated, so that the resulting upper 
bound on the risk is reliable. Our construction and proofs (in the appendix) closely follow 
the ones in (Cesa-Bianchi et al., 2004), using large deviation results for [/-statistics (see 
Clemengon et al., 2008, Appendix) instead of the Chernoff bound. 

4.1. Risk Analysis for Convex losses 

If the loss function <fi is convex in its second argument and y is convex, then we can use 
the average hypothesis h = ^j- Y^=c n h>t-i- It is easy to show that h achieves the desired 
bound (the proof is in the Appendix), i.e. 

F(K(h) > M n {Z n ) + e) ^ 



+ 1 



cxp 



(cn — l)e 
64 



+ Inn 



(19) 
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4.2. Risk Analysis for General Losses 

Define the empirical risk of hypothesis h t on {z t +i, ■ ■ ■ , z n } as lZ{h t ,t + 1) 

/ _ ,\ -i n 

The hypothesis /i is chosen to minimize the following penalized empirical risk, 

h = argmin (jz(h t ,t + 1) + c s (n - t)), (20) 

c„- l^t<n— 1 

where 



1 2(n - c n )(re - Cn + 1) 
p,(x) = \l — In . 



Notice that we discard the last two hypotheses so that any TZ(ht,t + 1), c n — 1 ^ t ^ re — 2 
is well defined. The following theorem, which is the main result of this paper, shows that 
the risk of h is bounded relative to M. n . The proof of Theorem 9 is in Appendix E. 

Theorem 9 Let ho, ■ ■ ■ ,h n be the ensemble of hypotheses generated by an arbitrary online 
algorithm A working with a pairwise loss £ which satisfies the conditions given in Theorem 1. 
Ve > 0, if the hypothesis is chosen via (20) with the confidence 5 chosen as 



(re - c n )e 2 
64 



5 = 2(re — c n + 1) exp 
then, when n is sufficiently large, we have 



f (ere - l)e 2 , 1 
exp|-^^ + 21nre|. 



5. Application: Online Algorithms for Bipartite Ranking 

In the bipartite ranking problem we are given a sequence of labeled examples z% = (xt,yt) € 
]R d x {— 1, +l},t = 1, • ■ ■ ,n. Minimizing the misranking loss ^ ran k under this setting is 
equivalent to maximizing the AUC, which measures the probability that / ranks a randomly 
drawn positive example higher than a randomly drawn negative example. This problem 
has been studied extensively in the batch setting, but the corresponding online problem has 
not been investigated until recently. In this section, we investigate two online algorithms, 
analyze their relative loss bounds and combine them with the main result to derive risk 
bounds for them. 



5.1. Online AUC Max with Infinite Buffer (OAM-I) 

Recently, Zhao et al. (2011) proposed an online algorithm using linear hypotheses for this 
problem based on reservoir sampling, and derived bounds on the expectation of the regret of 
this algorithm. Zhao et al. (2011) use the hinge loss (that bounds the 0-1 loss) to derive the 
regret bound. The hinge loss is Lipschitz, but it is not bounded and therefore not suitable 
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for our risk bounds. Therefore, in the following we use a modified loss function where we 
bound the Hinge loss in [0, 1] such that 

i{f, z t , Zj ) = ${{y t - Vj )/2, f(xt) - f(xj)) 

where eft is defined in Remark 4. Using this loss function together with Theorem 9 all 
we need is an online algorithm that minimizes M n (or an upper bound of M n ) and this 
guarantees generalization ability of the corresponding online learning algorithm. To this 
end, we propose the following perceptron-like algorithm, shown in Algorithm 1, and provide 
loss bounds for this algorithm. Notice that the algorithm does not treat each pair of 
examples separately, and instead for each Zt it makes a large combined update using its loss 
relative to all previous examples. Our algorithm corresponds to the algorithm of Zhao et al. 
(2011) with an infinite buffer, but it uses a different learning rate and different loss function 
which are important in our proofs. 



Initialize: wq = 0; 
repeat 

At the i-th iteration, receive a training instance Zt = (xt,yt) £ M d x {— 1,+1}. 
for j <— 1 to t — 1 do 

[ Calculate instantaneous loss ij = £(w t -i, z t , Zj). 
end 

Update the weight vector such that 

1 £=i 

w t = w t -i + 7— r X, i^ 31 * ~ 

until the last instance; 

Algorithm 1: Online AUC Maximization (OAM) with Infinite Buffer. 



Theorem 10 Suppose we are given an arbitrary sequence of examples Zt = (xt,yt),t 
1, • • • , n, and let u be any unit vector. Assume max||sct|| ^ R and define 



n _. t— 1 n 1 

^ t - 1 ^ 3 ^ t - 1 

t=2 j=l t=2 



t-1 



u 



where 



= \^vi ■ 



7- (u, -(y t - Vj)(xt - Xj)) 



. That is, M* is the cumulative average hinge loss u suffers on the sequence with margin 
7. Then, after running Algorithm 1 on the sequence, we have 



M < 



\/AR 2 + 2 + yf^W 



7 
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When the data is linearly separable by margin 7, (i.e. there exists an unit vector u such 
that £j = 0,Vt ^ n,j < t), we have M* = and the bound is constant. 

Proof [Proof of Theorem 10] First notice that wq = w\ = and we also have the following 
fact 



1 



7 - (u,-(y t - yj)(x t - Xj)) 



1 



7- <«. 7;(vt-yj){xt -xj)), 



which implies that when yt 7^ yj, 



(u,yt(x t -Xj)) = (u,-(y t - yj )(x t - xj)) > 7-^(11). 
On the other hand, when yt = yj, then l l - = 0. Thus we can write 



(21) 



(w t ,u) = (w t -i,u) + — —^2£j(u,yt{x t - Xj)} 

5=1 

t-i t-i 
> (wt-u u) + — ^4(7 - l](u)) = (w t - lf u) + ^ry 



5=1 

i-l 



t-1 



5=1 5=1 
t-i „ t-i 



(«7t,t») ^ E 

t=2 



7 



5=1 5=1 



7M-M*. 



On the other hand, we have, 



1 

5=1 



(22) 



t-i 



w t 



w t - 



- 1 !! 2 + FTi E 4(™*-i '^(^ - x 5-)) + 



5=1 

t-1 



1 

— ^tjytixt-Xj 



2 / t-i 



5=1 
't-i 



5=1 

^. > => {w^y^xt - Xj)) < 1) 



|™ t _i|| 2 + (4R 2 + 2) 



1 *=* 

5=1 



\w n f < (4i? 2 + 2)£ 



t=2 



1 * _1 

5=1 



(4# 2 + 2)M 



(23) 
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Combining (22) and (23), we have (7M - M*) 2 ^ (AR 2 + 2)M, which yields 



M < 



7 M* + (2R 2 + 1) + x /(2i? 2 + l)( 7 M* +2R 2 + 1) 



^ jM* + {AR 2 + 2) + ^7(2^2 + 1) 7 M* / V4i? 2 + 2 + ^tAF 



7 



We therefore get the risk bound for the proposed algorithm as follows. 

Theorem 11 Let Wo,-- - ,w n —i be the ensemble of hypotheses generated by Algorithm 1. 
Ve > 0, if the hypothesis w is chosen via (20) with the confidence 5 chosen to be 



5 = 2(n — c n + 1) exp 



(n - c n )e 2 
64 



then the probability that 



n — c r , 



'VAR 2 + 2 + 



7 



+ e 



zs at most 



^128^^/5 



+ 1 +1 



(cn — l)e 2 

exp < h 2 In n 

1 1 256 



Proof [Proof of Theorem 11] By (23), we can easily see that \\wf\\ ^ y n(AR 2 + 2),t = 
1, • • • ,n, therefore we have ||i0<|| • ||a:|| ^ R 2 \^5n, Vi ^ n. Therefore, we can take the 
hypothesis space to be 

rl = {w£M d : max \(w,x)\ < i? 2 \/5n}. 

By (4), the covering number can be calculated. On the other hand, from the definition in 
(2), it is easy to see that M n ^ M/{n — c n ). Finally, combining Theorem 9 and Theorem 10 
concludes the proof. ■ 



5.2. Mistake Bound for Perceptron 

Interestingly, we can apply our proof strategy in Theorem 10 to analyze the Perceptron algo- 
rithm in the inseparable case. This recovers the best known bound in terms of the one-norm 
of the hinge losses (given by (Gentile, 2003, Theorem 8) and (Shalev-Shwartz and Singer, 
2005, Theorem 2)), but using a simple direct proof. 
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Theorem 12 (Gentile, 2003; Shalev-Shwartz and Singer, 2005) Let (x\, yi), ■ ■ ■ ,(x n ,y n ) 
be a sequence of examples with \\xi\\ ^ R. Let u be any unit vector and let 7 > 0. Define 
the one-norm of the hinge losses as 

n 

D 1 = ^2i t , £ t = [-y-y t (u,x t )] + . 
t=i 

Then the number of mistakes the perceptron algorithm makes on this sequence is bounded 
by 

R + ^/Wi 

7 

Proof Let mt = \gn(u>fx t )^yt so that the total number of mistakes is M = ^2 t m t . Then, 
as usual, the upper bound is ||iUn|| 2 ^ R 2 M. On the other hand, using the fact that 
£ t = [7 — yt(u,x t )] + ^ 7 — yt(u,Xt), which implies yt(u,x t ) ^ 7 — It, we have the lower 
bound 

{w t +i, u) = (w t , u) + y t {x t , u)m t ^ (w t , u) + (7 - £ t )m t 

= {w t , u) + jm t - £ t m t ^ (w t , u) + jm t - £ t (".' m t ^ 1) 



(w n , u) ^ 7TO( - 1 
t=l t=l 



7M - Dx. 



(24) 



Combing the upper bound R 2 M with (24), we get (7M — D{) 2 ^ R 2 M. Solving the 
quadratic equation, we have 



M < 



1 

2^2 

l r 
-2 



27!?! + R 2 + V / 47i? 2 J Di + R 4 



2 7 2 



27D1 + 2R 2 + v / 4-fR 2 D 1 



R 2 + jDt + RjDi 



7 



5.3. Online Projected Gradient Descent for bipartite ranking 

We start by reviewing the online learning problem with univariate loss functions under the 
framework of online convex optimization (OCO). We are given a convex set K and at each 
step t, the algorithm selects a hypothesis Wt-x G K. Nature then reveals a convex loss 
function ft and the algorithm suffers a loss ft(wt-i). The goal of the online learner is to 
perform well comparing to the best w* which is obtained as if the whole data sequence 
is observed beforehand. More formally, we wish to develop algorithms that achieve a low 
value of regret R(T) after round T, which is defined as follows: 

T T 

R(T) = Y J ft(^t-i)- infV /*(«>). 

t=l t=l 

One algorithm that has performance guarantees is the Online Projected Gradient Descent 
algorithm, which consists of the following three steps: 
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1. Choose a learning rate n. 

2. Choose wq to be an arbitrary point in the convex set K. 

3. For all t = 1,2,- •• ,T, 

w t +\ = V K (x t - rf7f t {wt-i)), 
where Vk is the projection operator. 

The following theorem (Zinkevich, 2003) shows that this algorithm achieves 0(VT) regret. 

Theorem 13 Assume that K is bounded, closed and non-empty. Let 

D = max \\wq — w\\. 
weK 

Assume ft is convex and V ft exists for all t, and define 

G= max [|V/ t (ti;)|| 
te[T],weK 

to be the maximum I2 norm of the gradient of any ft in the set K . Choose r\ = D jG\pT ', 
then the regret of the Online Projected Gradient Descent algorithm after time T is at most: 

R(T) sC GDVT. 

Next, consider applying this algorithm to the bipartite ranking problem, where the hy- 
pothesis w is restricted to reside in a convex set K. Suppose ||:E(|| ^ R and max me x \\ w \\ ^ 
U, we define the normalized hinge loss as follows 

£hinge(w,Zi,Z 2 ) = 1 + \ RU i 1 ~ w T {x 1 - x 2 ){yi ~V2)] + , (25) 

which is convex in w and bounded in [0, 1]. In each step, nature selects the following convex 
loss function 

t-i 



ft(w t -l) = j^jYj^hingeiWt-l, Z t , Zj) 



which is also a convex function because it is a convex combination of convex functions. 
Thus, we can bound the sub-gradient as 



t - 1 1 + ARU 



t-i 



J2( x t ~ Xj)(y t - yj) 
i=i 



4R 1 
^ 1 + 4RU v U' 



Matching the terminology in Theorem 13, we have D = U and G = 1/U. It is easy to see 
that the Lipschitz constant of (25) is also upper bounded by 1/U. Therefore, we have the 
algorithm for online bipartite ranking given in Algorithm 2. 

From Theorem 13, we see that the regret of Algorithm 2 is bounded by ^fn. Let M* 
denote inf^, ^2 ft(w), i.e., the online loss of the optimal w. Using (19), we have the following 
theorem 
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Initialize: wq = and r) = 
repeat 

At the t-th iteration, receive a training instance Zt = (xt,yt) £ IR d x {— 1,+1}. 
Update the weight vector such that 



w t = Pk 



1 1 t_1 

Wt - 1 ~ l + ARU ' r^r ^ " yj ^ Xt ~ Xj) ' 14 

J'=l 



s (u) t _i,z t ,Zj)>0 



until i/ie fast instance; 



Algorithm 2: Online Projected Gradient Descent for Bipartite Ranking. 



Theorem 14 Let wq,-- - ,w n -i be the ensemble of hypotheses generated by Algorithm 2 
and let 



then the probability that 



is at most 



w 



n — c 



j n— 1 



i=c„ 



n - c r , 



+ 1 +1 



exp 



(era — l)e 2 
64^ 



+ lnra 



Notice that the above bound is worse than the one in Theorem 11 when the data is linearly 
separable. In the inseparable case, when the optimal cumulative loss M* = 0(n), it yields 
a better bound. 



6. Risk Bounds for Algorithms with Finite Buffers 

A natural criticism is that Algorithm 1 and 2 are not real online algorithms due to the fact 
that the entire sample is stored and at each iteration t, the update requires 0(t) time while 
online algorithms should have 0(1) time per step. 

To make it a real online algorithm, one can constrain the number of updates at each 
iteration. The idea is that at the t-th iteration, instead of keeping all previous t— 1 examples, 
we keep buffer Bt, whose cardinality can not exceed a predefined size \B\, that has a sample 
of the history. We call this type of online bipartite ranking algorithm OAM with finite 
buffer. 

One realization of this idea is using the "reservoir sampling" techniques from (Zhao et al., 
2011) (Random OAM) where the buffer Bt is maintained via reservoir sampling. Zhao et al. 
(2011) gave a bound on the expectation of the cumulative loss C = Y^t Ylj Translating 
their bound to our notation we get E[M] = M* + 0(y/n) where the expectation is over 
randomly sampled instances in the buffer. However, when the data are linearly separable, 
the cumulative loss given by this bound grows as 0(y/n) which is worse than the bound we 
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provided. In principle, one could turn the results of Zhao et al. (2011) into a high proba- 
bility bound on M using the Chebyshev's inequality and then use Theorem 9 to analyze its 
risk. However, this does not yield exponential convergence as above. Therefore, a natural 
question is whether we can provide similar analysis for OAM with finite buffer. The answer 
is positive. 

In the following, we provide the complete analysis. We give the generalization bounds 
for online learners with the finite buffer. Let us redefine the sample statistic as 

M n { z-) = J±—Y jM f{Z% Mf{Z i ) = -^— Hht-uz uZj ). (26) 

n Cn t= Cn iat ~ ll jm-i 

The difference is that for each hypothesis ht-i, the performance is evaluated on Xt and the 
examples in the buffer B t , which is a subset of the previous examples. Throughout this 
section, we assume a sequential buffer update strategy, or First In First Out (FIFO), as 
this simplifies the analysis. We believe that the same results hold for other random buffer 
strategies, e.g., the reservoir sampling, but leave this for future work. At iteration t, if 
Bt-i already hits the maximum size \B\, we substitute the oldest example in Bt-\ with z t ; 
otherwise zt is added to Bt-i- We can extend Theorem 1 as follows. The proof is similar 
and is provided in Appendix G for completeness. 

Theorem 15 Assume the preconditions in Theorem 1 hold and \Bt\ = (t — 1) A \B\ where 
\B\ is a predefined upper bound for the buffer size. Then, Vc > 0, Ve > 0, we have for 
sufficiently large n 



/ (|B LmJ |-l)f 2 

exp < h In ? 

y \ 64 

(27) 



Similarly, using the same technique of extracting a single hypothesis from an ensemble 
as in Section 4.2, we have the following theorem 

Theorem 16 Assume the preconditions in Theorem 1 hold. Ve > ; if the hypothesis is 
chosen via (20) with the confidence 5 chosen as 

(re - c n )e 2 
64 



6 = 2(n — c n + 1) exp 
then, when n is sufficiently large, we have 

p(*<s>>^ + «)<2[tf(«, iS ^i + 1 



exp < h 2 m n 

' 256 



Thus, we can see that the buffer size must grow faster than ln(n). We then introduce 
Algorithm 3, the finite buffer analog of Algorithm 1. 

Algorithm 3 is very similar to the algorithm in (Zhao et al., 2011) where the random 
buffer strategy is substituted with FIFO. We obtain the following mistake bound. 
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Initialize: wq = 0, Bq = 0; 
repeat 

At the t-th iteration, receive a training instance zt = (xt,yt) G H d x {— 1,+1}. 
Update the weight vector such that 

w t = wt-x + ,J~ i E $m{xt - Xj) 
\ot-x\ .r^ 

Update the buffer using FIFO such that Bt = Bt-x Uzt; if \Bt\ > \B\, remove 
z t-\B\+x ou t °f t ne buffer, 
until the last instance; 

Algorithm 3: Online AUC Maximization (OAM) with Finite Buffer. 



Theorem 17 Suppose we are given a sequence of examples Zt,t = 1, • • • ,n, and let u be 
any unit vector. Assume max||sct|| ^ R and define 



n 



t=2 



B t -x\ 



E 4H 

jeBt-i 



n 1 



t=2 



B t -x\ 



E 



Then, after running Algorithm 3 on the sequence, we have 



'V4i2 2 + 2+ v/rMl' 



The proof is almost identical to that of Theorem 10 and we include it in Appendix F for 
completeness. 

Similarly, using the model selection approach describe before, we have 

Theorem 18 Let wq,-- - ,w n —i be the ensemble of hypotheses generated by Algorithm 3. 
Ve > 0, if the hypothesis wq is chosen via (20) with the confidence 5 chosen to be 



5 = 2(n — c n + 1) exp 



(re - c w )e 2 
64 



then the probability that 



n(w B ) > 



re — On 



V4i? 2 + 2 + y^W* 



+ e 



is at most 



^ 32gVjjn 



+ 1 +1 



(\B cn \-l)e 2 

exp < h 2 In re > . 

1 ^ 256 
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Initialize: wq = and rj = -^=; 
repeat 

At the t-th iteration, receive a training instance Zt = (xt,yt) G TR d x {— 1,+1}. 
Update the weight vector such that 



w t = Pk 



Wt ~ 1 ~ aru+1 'tn S ^ -^o-^m^. 



(tl! t _l,Z(,Zj)>0 



until i/ie fast instance: 



Algorithm 4: Online Projected Gradient Descent for Bipartite Ranking with Finite 
Buffer. 



We have similar extension for Algorithm 2, as described in Algorithm 4. It has the same 
risk bound as in Theorem 13 where M* is substituted by MS. 
Finally we relate Mg(w) to M*(w), which is defined as 



M*(w) 



n 



I n—l ^ t— 1 

_ c ^ t — 1 ^ 

t=Cn J=l 



[W,Z t ,Zj J 



It is easy to see that for any fixed w, M*(w) is an unbiased estimator of lZ(w). Therefore, 
if Mg(w) is close to M*(w) with high probability, we can say that 1Z{wq) is close to TZ(w) 
(Notice that w can be arbitrary) with high probability. We have the following lemma, 

Lemma 19 Suppose the pairwise loss £ satisfies the conditions given in Theorem 1. As- 
suming the sequential updating rule for the buffer, then Ve > 0, we have 



sup 

w 



1 



M* B (w) 



1 



n — 2 ~ ' n — c n 
Proof To prove this, define Q(w) as 
1 1 



M* cn (w) 



8Lip(4>) 



-c 2 (l-c) 2 ne 2 



n(w) 



n-2 
1 



M* B (w) 



n-l 



n 



(n 



2 t=2 \ Bt \jm 



n 



n-l 

E 



t-i 



[W,Z U Zj t 



Suppose the sequential buffer strategy is used where Bt = {x t -i,x t -2, ■ ■ ■ , x t ig\ + 1}. We 
next compute the variation of Q(w) when changing any of its n random variables. For both 
terms, there are two situations when one substitutes x% with x' i: i.e., when t = i and t > i. 

• For the first term, when t = i, its variation is bounded by l/(n — 2). When t > i, 
as we use FIFO, Xi can only be kept in the buffer for B rounds, thus the variation is 
bounded by l/(n — 2). 

• For the second term, when t = i, the variation bounded by l/(n — cn). We have 
previously shown in (11) that changing x, when % > t alters the second term by 1/cn. 
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Consequently, Q(w) is bounded by 2/(c(l — c)n) when one variable is changed. It is easy 
to see E[n(iu)] = 0. By McDiarmid's inequality, we have 

F(n(w) ^ e) ^ exp- c2(1 - c)2ne2 . 
Using the covering number technique again, we have 

P(supf)(™) > e) N (h, OT .* 1 e-^^ 1 -^ 2 ^ 2 . 

w V 8Li p(<p) / 



7. Application: Online Metric Learning 

In the past decade, metric learning has become an active field in machine learning with nu- 
merous applications in information retrieval, classification, etc. In this section, we consider 
online learning algorithms for supervised metric learning. Generally speaking, supervised 
metric learning seeks to find a Mahalanobis distance metric that makes instances "agree 
with" their labels. The intuition is that under the desired metric, examples that share the 
same label should be close while ones from different labels should be far away from each 
other. The metric is parameterized by a positive semi-definite matrix A such that for any 
two example Zi , Zj , we have 

d A (Zi,Zj) = yj (Xi - Xj) T A(Xi - Xj). 

Under the batch setting, recent work derived the generalization bounds for supervised metric 
learning (Jin et al., 2009; Cao et al., 2012; Belief and Habrard, 2012). Several online metric 
learning algorithms have been proposed (Davis et al., 2007; Jain et al., 2008; Jin et al., 
2009; Kunapuli and Shavlik, 2012); these analyzed to provide regret bounds but to date 
the generalization performance of these algorithms has not been analyzed, possibly because 
no tools existed to provide online-to-batch conversion with pairwise loss functions. In this 
section, we provide such an analysis. 

We consider the hypothesis space to be the vector space of symmetric semi-definite 
matrices of size d x d, equipped with the inner product, (X, Y) := Tr^X^Y). ||X|||i = 
(X,X) denotes the Frobenius norm of matrix X. Following Jin et al. (2009), we will work 
with the following pairwise loss function 

£(A,Zi,Zj) = g (yij [l - [xi - Xj) T A(xi - xj)]) , 

where g is a normalized version of the hinge loss and y^ = 1 if yi = yj and —1 otherwise. 
With this setting, examples in the same class must have distance to obtain zero loss and 
examples in different classes must have distance larger than 2 to yield zero loss. 

Before further development, we first state and prove the following theorem for online 
gradient descent over matrices. Its proof is similar to the case where the hypothesis space is 
IR d (Zinkevich, 2003), but we include a proof for completeness. More sophisticated analysis 
for learning with matrices is provided by Kakade et al. (2012). 
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Theorem 20 Assume that K C is convex, closed, non-empty and bounded such that 

sup || A — B||jr < U. 

A,B 

Assume that at round t we are working with a convex loss function It : — > JR + such that, 

\\V£ t (A t )\\ F ^D. 
Consider an online learner with update rule 

A t+1 =V K [A t -rjV£ t (A t )}, 

where Vk is the projection operator. If we set the learning rate rj = jj\ h, we have 



T T 

5>(At) - urf J>(B) ^ UDy/T. 

t=\ t=i 

Proof Since K C is convex, closed, non-empty and bounded subspace of a Hilbert 
space, we have (Rudin, 2006), VA,B € Sj", 

||7MA)-^(B)|||^||A-B|||. 

For an arbitrary B G K, we have 

||At + i — B||| — \\A t — B||| 

= \\V K [A t - r/V4(A t )] - BUI - ||A t - B||| (-.- \\P K (X.)f F < ||X|||) 
< ||A t - B - r]V£t(A t )\\ 2 F - \\A t - B||| 
= ||»/V^(A t )|||.- 277(V€ t (A t ),A t -B) 



which gives 



(W t (A t ), A t - B) ^ ±- (||A t+1 - B||| - ||At - B||| + n 2 \\Vl t (A t )\\ F ) . 



Therefore, 

T 



J>(A t ) - J>(B) < ^{W ( (A t ), At - B) 
t=i t=i t=i 

T 



2n 
1 t=i 

< — lT 2 + ^D 2 T. 
2?7 2 

Setting the learning rate r\ = %\/ ^ yields the result. 



< ^E(H A m- B||| -||At -B|||) + ^ 2 T 
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For the metric learning, at each round the loss function is 

MA) = ^ I E4(A), 

i=i 

where 

4(A) = [1 - y tj {l - {x t - Xj) T A(x t - Xj ))) + 

= [l-yt j (l-(A,X tj ))) + , 

where ytj G {+1,-1} and ~K t j = (x t — xj)(x t — xj) T . It is easy to see that £*• is a convex 
function of A. 

Next we choose the convex set K to be 

K = {A : A G S£,||A|| F «S U}. 

We assume sup t \\x\\2 ^ R. To bound it hi [0,1], we redefine scaling it by a factor of 
2+uK 1 • ^° u * mze Theorem 20, we need to bound the subgradient of as follows, 



V4(A)|| ^ 



1 



2 + UR 2 



\\VtjX+j\ 



R 2 



1 

2 + UR 2 ^ U' 



Initialize: r] = and Ao to be any PSD matrix with ||Ao||f ^ U. 
repeat 

At the t-th iteration, receive a training instance Zt = (xt,yt) G JR d x {— 1,+1}. 
Update the weight vector such that 



A t = V K 



until the last instance: 



1 1 * _1 



)>o 



Algorithm 5: Online Projected Gradient Descent for Metric Learning. 

Finally, as in previous results, define M* = infe ^t(B). Using (19) we bound the risk 
of Algorithm 5 with the following theorem. 

Theorem 21 Let Ao, • • • ,A n _i be the ensemble of hypotheses generated by Algorithm 5 
and let 



n-l 



n — c, 



t=Cn 



then the probability that 



TZ(A) ^ 



n — On 



is at most 



, 32i? 
21 + 1 +1 



exp 



[M* + y^) + e 
(cn — l)e 2 



64 



+ Inn 
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The key computational step is to project the matrix to St. Jin et al. (2009) showed an 
efficient way to perform the projection. Notice that Theorem 21 also applies to the online 
learning algorithm proposed in Jin et al. (2009) with proper normalization. In general, we 
believe our results are applicable for all online metric learning algorithms with proved regret 
guarantees. 

8. Conclusion and Future work 

In this paper, we provide generalization bounds for online learners using pairwise loss func- 
tions and apply these to bipartite ranking and supervised metric learning. These are the 
first results to provide online-to-batch conversion for pairwise loss functions and as we 
demonstrate, they are applicable to multiple problems. 

There are several directions for possible future work. From an empirical perspective, 
although the random Online AUC Maximization (OAM) is simple and easy to implement, 
it seems that it does not maintain buffers in an optimal way. Intuitively, one might want to 
store important examples that help build the correct ranker instead of using a random buffer. 
We are currently exploring ideas on building a smart buffer to improve its performance. 

From the theoretical point of view, one direction is to improve the current bounds 
to achieve faster convergence rates. Another direction is deriving tighter mistake bounds 
for random OAM. Finally, our buffered based results require a buffer size of O(logn) to 
guarantee convergence. It would be interesting to investigate whether this is necessary, or 
alternatively improve the results to show convergence for constant size buffer. It is also 
interesting to extend our results to dependent data, for example, assuming the data to be 
stationary mixing sequences (Agarwal and Duchi, 2013) or utilizing new tools such as the 
sequential Rademacher complexity (Rakhlin et al., 2012). 
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Appendix A. Complete Proof of Claim 1 

Proof [Proof of Claim 1] The required probability can be bounded as follows. 



z n ~r> 
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We next show that for sufficiently large n, 
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which combined with (28) implies (9). To begin with, we first show that the corresponding 
random variable has mean zero 
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Thus, we can use Chebyshev's inequality to bound the conditional probability as follows 
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To bound the variance, we resort to the following Theorem (see Devroye et al., 1996, The- 
orem 9.3) 

Theorem 22 Let X\,--- ,X n be independent random variables taking values in a set A, 
and assume that f : A n — > IR satisfies 



sup |/(sci,--- ..r,.- . 

Xi,X2,X n ,x' 



x n ) - f(x 1 , • • • ,x',--- ,x n )\ ^ a VI ^ i < n. 



Then 



1 n 

VarifiX,,--- ,X n ))^-J24- 



i=l 



To bound the variance, we first investigate the largest variation when changing one 
random variable with others fixed. From (8), it can be easily seen that changing any 
of the £j varies each Et[M(], where t > j by at most by l/(t — 1). Remember we are 
only concerned with Et[JVft] when t ^ c n . Therefore, we can see that the variation of 
n } c Ylt=cn ^t[Mt] regarding the jth example £j is bounded by 
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n — c r . 
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_t=(jVc n +l) 

Thus, by Theorem 9.3 in (Devroye et al., 1996), we have 
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ft— 1 ^ 
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4 ^ * ^ 4c 2 n' 
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(29) 



Thus, whenever e 2 c 2 n > 2, the LHS of (10) is greater or equal than 1/2. This completes 
the proof of Claim 1. ■ 



Appendix B. Proof of Lemma 5 

Proof [Proof of Lemma 5] To bound the probability, we use the McDiarmid's inequality. 

Theorem 23 (McDiarmid's Inequality) Let X±, ■ ■ ■ ,Xn be independent random vari- 
ables with X k taking values in a set A k for each k. Let (j) : [A\ x • • • x An) — > IR be such 
that 

sup • • • ,x N ) - <j>{x\, ■ ■ ■ ,x k -i,a/ k ,x k +i, ■ ■ ■ ,x N )\ < c k . 

Then for any e > 0, 

o 2 / N 2 

P{^(xi,--- ,x N )-TE(f>(xi,--- ,X7v)^e}^e- 2e /^=i c S 

and 

P{|0(xi,--- ,x N )-JE^(xi,--- ,x N )\ > e} < 2e- 2e2 /Eti^. 
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For any fixed / £ %, we have 



IE 



21 



1 

= E ( E fe E *W/> &)] - ^z^zW, z, Zj )]) = 



Now, Lt(f) is a function of 2(t — 1) variables with each affecting its value at most by 
d = l/(t - 1),» = 1,2,- •• ,2(t- 1). Thus, we have Ei=i _1) c i = Finally, using the 
McDiarmid's inequality, we get 

~2J*,B«~2>« 0&t(/) > e) < exp {-(* - l)e 2 } . 



Appendix C. Proof of Lemma 6 

Proof [Proof of Lemma 6] From the definition of Lt and the assumption on (j) we have 
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Appendix D. Proof for the Risk Bound of Convex Losses in Section 4.1 
Proof Using Jensen's inequality, we have 
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Combining with Theorem 1, we have 



■(K(h) > M n {Z n ) + e) ^ 
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Appendix E. Proof of Theorem 9 

Proof [Proof of Theorem 9] The proof is adapted from the proof for Theorem 4 in (Cesa-Bianchi et al. 
2004). The main difference is that instead of using the Chernoff bound we use a large de- 
viation bound for the [/-statistic as follows. 

Lemma 24 (see Clemengon et al, 2008, Appendix) Suppose we have i.i.d. random vari- 
ables Xi, ■ ■ ■ ,X n G X and the U— statistic is defined as 



1 n 2 n 



n(n — 1) 



where the kernel q : X x X ^ TR is a symmetric real-valued function. Then we have, 

¥(\U n - 1E[U n ]\ ^ e) < 2exp{-(n - l)e 2 }. (30) 
Therefore, by (30), we have 

F(\n(ht,t + 1)-H(ht)\ >c) ^2exp{-(n-t-l)e 2 }, 

or equivalent ly, 

p [ n(ht,t + 1) - n(ht) > \ / — : — - in - | v s. 



n-t-l 5 



(31) 
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By the definition of c$ and (31), one can see that 

F(\K(ht,t + l)-n(h t )\ >c 5 (n-t))^- — . (32) 

(n - c n )(n - c n + 1) 

Next, we show the following lemma, 

Lemma 25 Let ho, ■ ■ ■ , h n —\ be the ensemble of hypotheses generated by an arbitrary online 
algorithm A working with a pairwise loss £ which satisfies the conditions given in Theorem 1. 
Then for any < 5 ^ 1, we have 



F K(h) ^ min (K(h t ) + 2c 5 {n - 1)) < 5. (33) 

\ c n — l^t<n— 1 J 

Proof [Proof of Lemma 25] The proof closely follows the proof of Lemma 3 in (Cesa-Bianchi et al., 
2004) and is given for the sake of completeness. Let 

T* = argmin (K(h t ) + 2c 5 {n - t)) , 

Cn~ l^t<7l— 1 

and h* = hx* is the corresponding hypothesis that minimizes the penalized true risk and 
let 1Z* to be the penalized empirical risk of h?*, i.e. 

K* =U(h T *,T* + 1). 

Set, for brevity 

K t = K(h t ,t + l), 

and let 

T = argmin (TZ t + cs(n - t)), 

c n — l^t<n— 1 

where h defined in (20) coincides with h^. With this notation, and since 

TZ f + c 5 (n - f) < 7T + c 5 {n - T*) 
holds with certainty, we can write 
w(ll{h) > K(h*)+£) 

[n(h) > K{h*) + £, K f + c s {n - f ) < K* + c s {n - T*) 

^ P(K(ht)>K(h*)+£, TZt + c s (n-t) ^K* + c 5 (n-T*)) 

t=Cn— 1 

where £ is a positive-valued random variable to be specified. Now if 

n t + c s (n -t)^TZ* + c s (n - T*) 
holds, then at least one of the following three conditions: 
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must hold. Therefore, for any fixed t, we can write 

P (lZ{h t ) > K{h*) + £, Kf + c 5 {n -t)^K* + c 8 (n - T*) 

< P (ft* < U{h t ) - c s (n - i)) + P (K* > n(h*) + c 5 (n - T*) 
+ F(K(h t )-K(h*) <2c s (n-T*),K{h t ) >K{h*)+£). 
The last term is zero if we choose £ = 2cs(n — T*). Hence, we can write 
P (n(h) > K(h*) + 2c s {n - T*)) 
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Therefore, we know that 
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The next step is to show that with high probability min (TZ(hf) + 2cs(n — £)) is close 
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to M™. To begin with, notice that 
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where the last equality holds because Y17=l 1 \AA ^ 2\/n — £ — 1 (see Cesa-Bianchi et al. 
2004, Sec. 2.B). Define 



Mr, 



-. n— 1 

— ^M t (Z*). 

- m ^ — ' 



n — m 



t=m 



From Theorem 1, one can see that for each £ = c n — 1, • • ■ , n — 2, 
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Then, set for brevity, 

K t = M tn + 4 
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m h e. 
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Using the fact that if min(ai, 02) ?S min(6i, 62) then either a\ ^ b\ or 02 ^ 62 j we can write 
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which, in particular, leads to 



P I 11(h) >M n + 4j^— ln 2(n Cn + 1) +< 
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exp \ — h 2 In n 
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By substituting e with e/2 and choosing 5 as in the statement of Theorem 1, that is, 
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Appendix F. Proof of Theorem 17 

Proof [Proof of Theorem 17\ First notice that wq = w\ = and from (21), we also have 
(u, yt(xt — Xj)) ^ 7 — £j whenever yt ^ yj and £*■ = otherwise. Thus similarly, we have 



(W t ,u) = (W t -l,u) + — — r V tUu,y t (x t - Xj)} 
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On the other hand, we have, 
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I ||2 II ||2 , 
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Combining (36) and (37), we have {^M B - M£) 2 ^ (AR 2 + 2) Mb, which yields the desired 
bound. ■ 



Appendix G. Proof of Theorem 15 

Proof [Proof of Theorem 15] Again, we rewrite our objective 
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Thus, we can bound the two terms separately. The proof consists of four parts, as follows. 
Step 1: Bounding the Martingale difference 

First consider the second term in (39). We have that Vt = (JEt[M®] — M t )/(n — c n ) is a 
martingale difference sequence, i.e. IEjVt] = 0. Since the loss function is bounded in [0, 1], 
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we have \Vt\ ^ l/(n — c n ), t = 1, • • • , n. Therefore by the Hoeffding-Azuma inequality, 
J2t Vt can be bounded such that 
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1 n— 1 
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(40) 



Step 2: Symmetrization by a ghost sample E n 

Recall Mf and define Mf as 



Mf(Z*) = T ^— Y, nht~i,z t , Zj ), M f(Z t ) = ^— Y, Hh-uzt,^). (41) 
\Bt-i\ ~zr \"t-i\ .~Tr 



Claim 2 The following equation holds 
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(42) 



whenever (1 — c) n ^ 1/2. 

Notice that the probability measure on the right hand side of (42) is on Z n x S n . 
Proof It can be seen that the RHS (without the factor of 2) of (42) is at least 
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Since E=n^-pn E t [Mf] = ^(/i t _i), by Chebyshev's inequality 
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n-1 

E 

t=c n 



(43) 



To bound the variance, we first investigate the largest variation when changing one random 
variable £j with others fixed. From (41), it can be easily seen that changing any of the £j 
varies each TE t [M^], where t > j by at most by l/|0t|. 

To estimate the difference the variation of n j L e Y^t=c n ^[Mf 3 ] regarding the jth example 
£j , it is easy to see that since we are using the FIFO strategy, any example can only stay in 
the buffer for B round. Changing each example can only result in l/\B\ for E t [M"f]. Thus, 
the variation is bounded by 



n — Cn 



n — c n \B\ n — c n 
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Thus, we have 

Var | ^— Y JE t [M t B ] \ - 1 , 9 . (44) 

\n-c n ^ 1 / 4 4(1 -c) 2 n V 7 

Thus, whenever (1 — c) 2 n ^ 1/2, the LHS of (43) is greater or equal than 1/2. This com- 
pletes the proof of Claim 2. ■ 



Step 3: Uniform Convergence 

In this step, we show how one can bound the RHS of (42) using uniform convergence 
techniques, McDiarmid's inequality and L M covering number. Our task reduces to bound 
the following quantity 



/ n-l 



Define L^h^) = E t [Mf] - E 4 [Mf ]. Thus we have 
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To bound the RHS of (46), we start with the following lemma. 
Lemma 26 Given any function f £ % and any t ^ 2 

Pz»~27«,st~2J* OW) ^ e) ^ exp{-(|Bt| - l)e 2 } . 



(47) 



We have already proved the case when t ^ \B\. When, \B\ < t, L t (f) has a bounded 
variation of l/|i3(| when changing its 2\Bt\ variables. Applying McDiarmid's inequality, we 
immediately get the desired inequality. Using exactly the same reasoning as in Lemma 6-8, 
we get 



Lemma 27 For every 2 ^ t ^ n, we have 



snp[L t (h)]>e) ^Mln,——- 
heu J V 4Lip(<f>) 



Combining (48) and (46), we have 
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Step 4: Putting it all together 

We have 



(50) 
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